Dynamic recurrent neural networks

We survey learning algorithms for recurrent neural networks with hidden units and attempt to put the various techniques into a common framework. We discuss fixpoint learning algorithms, namely recurrent backpropagation and deterministic Boltzmann Machines, and non-fixpoint algorithms, namely backpropagation through time, Elman's history cutoff nets, and Jordan's output feedback architecture. Forward propagation, an online technique that uses adjoint equations, is also discussed. In many cases, the unified presentation leads to generalizations of various sorts. Some simulations are presented, and at the end, issues of computational complexity are addressed. This research was sponsored in part by The Defense Advanced Research Projects Agency, Information Science and Technology Office, under the title "Research on Parallel Computing', ARPA Order No. 7330. issued by DARPA/CMO under Contract MDA972-90-C-0035 and in part by the National Science Foundation under grant number EET-8716324 and in part by the Office of Naval Research under contract number N00014-86-K-0678. The author held a Fannie and John Alexander Hertz Fellowship. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of the Hertz Foundation or the U.S. government.